Description

Avoiding try_get_deltatable solves the issue.

I tested by running pipeline.py in a loop:

import os
import dlt
from dlt.destinations import filesystem

os.environ["RUST_BACKTRACE"] = "full"

num_resources = 5
resources = [
    dlt.resource([{"foo": "bar"}], name=f"r{n}")
    for n in range(num_resources)
]
pipe = dlt.pipeline(
    pipeline_name="delta_source",
    pipelines_dir="_storage",
    destination=filesystem("_storage"),
)

pipe.run(resources, table_format="delta")

for run in {1..100}; do python pipeline.py; done

Results:

• old impl with try_get_deltatable: 100 runs, 16 panics
• new impl: 100 runs, 0 panics

Old impl used try_get_deltatable because DeltaTable.is_deltatable() did not yet exist. It got added after I opened a ticket in the delta-rs repo.

Related Issues

Fixes #1808

Additional Context

• also brings airflow back to make dev to prevent mypy errors